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PREFACE 



Extension's productivity and acbountability C£(n 
be greatly advanced through, evaluation of its^ro- 
grams. This publication is mtended to help ExtensicAi 
administratcirSy jprogram leaders, and specialists to 
fulfill their program evaluation responsibilities, ^ 

Emphasis is placed bn maximizing t^ usefulne^^ 
of program evaluationji in decisionmakmg on pr^ 
grani priori ties ana^^[odifi<»tipns- / , ^ , " 

Harmonizdng^sd progranit;; evaluation aVail- 
^^ble resour;es.isjb chaUeogei program evato 
especially t]ie collection of evidence, c^n b^i cdstly. 
Analyzing impacts of Extension Programs presents 
optional le\ els of eviden^\wjh.vajrying i3egre^^ 
expense, _ 

Soine of thd ideas in this publication are yet to be 
verified, bi^t are presented to guide thinking about^" 
Extension program evaluation and to lead to tested 
principles pf evaluation. 



AdministratiVfe support for pjrep^ation of tfiils i 
publication was extended by C. A. Williams, Deputy 
Administrator, Program and ^Staff "Development, 
Extension Service, U.S. Department of .Agriculture.' 

The Extension Service pf the U.S. Department of 
Agriculture offers its programs to all eligible, 
p^sons regardless of race, color or. national origin, . 
and is an Equal Opportunity Employer, 

Cooperative Extension Work: U.S. Department of . 
Agriculture and Sfate Land-Grant Universities 
Cooperating. Issued April 1976. ^ 
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j/"; This pubUciitioir presents a; framework, guider.^^^ 
"'■Udubs; strategy,/ an methods for evaluating Extens 
education prograips.^^xtension^ are vieW^d/' v 

in'terms of seven levels of objectives and eyaimMo/ ' ^ 
. y evidencef (i) inputs^ ; (2) acuities, < 3) peopleA 
' involvement, ' (47) reactions, (5) change of knowl- . 
; edge, ^tUtiides, i5]kiUs;^^ ' • 

(6) practice chaiige^ and (7) end res^^ 

Levels' 1 and 2 characterize Ex^t^^^^^ >• 
Level 3 includes the people involved by^Eiterision 
* and the xiature of their inyolveme^^^ 

ihrpugh/^^^^ the responses by these |)eo and 
/ othcjrs^ and • 

dir^c^ to the lorig^erm^ 
;Extj^sionV . actions. ■ • ■/,m-'}:\^ 
/ ppie foregoing ley^^^ in/(a) the extent tdf : / 
i: which they can provide evidence of Extension's 
iinpafct>and j;b) the amouht^f resources required 
, fo^.bbtal^ 

^ p^pgr^^ becomes stronger in ascending the 

l^l^ye^s^ H^ obtaining evidence at higher levels ' 

; . generally requires . more evaluative resources. The . * 
> '/lev^^ of evidence chosen for ^ particular prbgram 

evaluation will vary 
-s- the na^re of the progra circumstances '-.."^ 

of it^ evaluation. Proxy indicates are suggested, in 
order to maximize strength of evidence in lower 
cost -a^ssments* of Extension's effeptiyeness. 
■^Progriri evaluations may be relied upbh to 
assist •d^isionmaking' to the extent that they provide 
"hj^^^ality evidence of accomplishment of pro- 
^am obj^ectives and identify Extension's extent of 
cojitributipn tp such accomplishments. 
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iE^MALYZING IMPACTS 
tJjFEXTENSibNvl 1 

P|?0GF{AMS 



By Claude F. Bennett 
Specialist, Educational Methodology and Evaluation 
Program and Staff Development 



i^TRODUCTIOM I 

"Are Extensibii^^ is a ques- 

tion ask^dfrequesntly by officials !at all levels of , 
Government, legislators, university adiriinis'traters, 
aid Exterision wo rlcers themselves. TTiis puBlicration 
Provides g^iidance in eyd^ 
I Extension bducatioa . programs.^ j 

. Judgments :about = program effectiveness will be 
./inade one way or anpther/ However, there is mount-- 
ing demand tby legislators, ;^^ " and 

administrator tharp^^^ 

strated- thrbiigh formal evaluations. TJ^ese de-. 
-mands; reinforce the desire by^ Extension staff to 
: obtain sound .evidence of to which Exten- 

sion programs are successful. Foirma/ evaluation ' 
:eptails :cqnscidus procedures for placing value on 
l^roj^rdrn^ according to (I yexplicit 
\ (2^ designs for collection and analysis of evidence. 
;:y?vy j^rogram evaluation is part of j tlidvpverall program 
redevelopment process, which incljUc[es:/(l) Identify- 
ing problems and selectinglong-rang^ ol^jectives;*-^ y 
■ (2 X specifying these pbjectives ^d the striategy; 
' activities, and budget designed td ach^v^ them| (3^^^^^^ 
conducting activities; (4) evalukting the prograin^|/ V 
strategy apd impact; and ('5) rising this evaluation ' 
along w^ other information inj subsequent progr^ 
development. ./ 

Impact evaluation is assessment of a program's 
effectiveness in achieving its ultimate' objectives ' i 
or ^assessment of relative effectiveness of twp or more 
programs ^. meeting common ultimate objectives,^" 



T}ie major purpose of program evaluation is to 
assist in leachmg decisions on future directions, . 
-design, 'and funding of programs.^ Decisions on 
whether programs should be terminated, curtailed, 
maintained, or expanded arp aided by program 
ii evaluations. ' : 

Such evaluations may also suggest reformulation 
of program objectives, strategy,- delivery organiza- 
tion, educational methodology, and intended;/' 
audiences. , , ' 

TJjis publication identifies seven broad categqfies 
of criterii which are useful in formally evialuating 
the effectiyeness^^ofJExtension programs and attempts 
to provide guidance in choosing evidence regarding 
these categories. * 

A CHAIN OF EVENTS IN ' 
EXTENSION PROGRAMS 

Figure- 1 shows a "chain of events" assumed to 
characterize*most programs of Extension education. 
Although the events selected oversimplify reality, 
they ^fovide a "mind-^hold'V on Extension programs, 
t The events, chart the behavior of ^bbth Extension 
and the people ^involved in its programs.* • 
!7' "/np«/^'*j(l9Wer left of fig. .1) are selected on 
the assumption that problem solution may require 
resource expenditures.' With these inputs, ''Activities" 
can be performed; e.g., publicizing programs or 
"putting "across" educational content. 

Activities "involve People'' (participants) who 
have "Reactions", i:e., some degree of interest 



^^Cooperative Extension education is defined herein as 
noncredit individual, group, and mass instruction directed, 
toward practical problem-solving. Usually co;iducted 
off-campuis and informally, Cooperative Extension ptogi^ams 
are an outreach of Land-Grant Universities and Colleges!:--" 
Cooperative Extension Service programs are generally 
mutually funded and directed by local, State and natiorial 
sources. See U.S. Department of Apiculture, National > 
Association of State Universities and Land Grant Colleges 

'Study Committee, A People and A Spirit, 'Fort Collins, 

. Colo., Colorado State University," 1968. 
/^'Scriven, Michael, "The M^hodology of Evaluation,'* , 
Perspectives of Curriculum Evaluation, Ralph Tyler, 
Robert Gagne, and Michael Scriven (eds.), pp. 39-83, . 
Chicago, 111., Rand McNally, 1967. \ 
Stufflebeam, Daniel L.,. "Toward a Scienop of ^Education 



^ Evaluation," Educational Technology S (July 1968), 
pp. 5-12. , 

^ Wholey, Jpseijli S.i John W. Scanlon, Hugh G. Duffy, 
James S. Fukumoto, and Leona M. Voght, Federal Evalua- 
tion Policy: An Overview, Washington, D.C, t^ban 
Institute; 1970., 

*Stufilebeam, DanieJ L., op. cit., and Joseph S. Wholey, 
op. cit.; see also Warner, W. Keith, "Feedback in Adminis- 
tration," yo«r/iafo^Ejc/e/w/o/i V (Spring 1967), pp. 35-46. 

* Several elements of the chain have been identified by 
Kirkpatrick; a|[^Suchman. ^ee Kirkpatrick,^-Donald L„ 
"Evaluation ofN^^aining," Tra/m'/i^ and Developrnent Hand^ 
boojf, Robert L. Craig and Lester R.'Bittel (eds.),jpp. ■ 
87-112, New York, McGraw-Hill, 1967, and Suchman, 
Edward A., Eyaluative^Research, New York, Russell Sage 
Foundation, 1967. , , . 



6. PRACTICE CHANGE 



5. KASA CHANGE 




3. PEOPLE INVOLVEMENT y 




INPUTS 



in,.and like of dMike for, th'e activities in which; 
: fliby are involved;^ (Reactions to activities depend 
\ pri both the activities themselves and^the values, 
; learning ability, and socill interrelationships of the 
l ^^ peoplej mvblved ) T^) the extent that participants' ' \ 
ilintef^st can be held,^t}iey may change tiieir fcnbwl- 
• edge; attitudes, skills,^^^^^ 

Wherfeas attitude denotes feelings fapproval or 
disapproval), aspiration Indicates tiie W of feelmgs^. 
in goal sejection or "choice among alternatives . ' ^ 
' - 'Practice Change" (adoption) refers to individual^ 
or collective application of acquired knowledge, 
•^attitudes, skills, and aspirations to vvjork.or life 
ji^styles.^ But^ practices are not usually adopted - * 
^for.tiieirpwn sake; certain benefits arfe anticipated 
to acchi^ from individual; and collejetive practices. 
Whatever benefits and consequences follow from . 1 
pracjices may be called "E/jrf Results.'- T)iese?- i ' 
resdt?i hpjtefolly, md tiie ultimate 

objective (s) of , Extension programs. - ' 

Before continuing, it shoujd be acknowledged : • 
, that individual or group, change may ne|t always i 
proceed strictiy in accordance with, thfe^above - i 
sequence of events. For ^^ample, tractions prpb- : 
ably occur prior to aifd durbg participation, as^^^^eU ' 
as after ijiv&lvement. Also, pxactice change^inay. r * 
occur befor^he attitude or knowledge change ■ , 
i intended by program objectives.* 

A HIERARCHY FOR PROiSRAM 
EVALUATION • ' : ■ ^. i «v ' 

In figure 2v the foregoing, chain, 6f events is con- ' 
verted into a hierarchy of objectivei arfcf evidence ' 
for program evaluations. Six leye^s/Qf^tput are 
^ based up)on inputs to Extension. 



° Initially, it matters litt!e|\v^e:thei^ participants a^^ 
interested in the educational .Content of , ac't^^^^^^ 
ample, some participants may- afeiirst a«6nd yalu^' ' .* 
discusaions on grain production or iititritipn l)ecausl tiley' 
enjoyj the social interaction;' all^d-.j^^^^ fV 
particularly interested in the subje^;:xinder 'di^cu^ 

" Of course, KASA change may siiiiply reinforc^ cxist- 
ing practices. Clientele jnay be expected^^tp exhibit 'PO ^ ' 
practice change, as in education' to prevent uSe of harmful 
drugs. ' ; 



At each level of the hierarchy, ^for plsmned) 
symbolizes an objective to be' reache^i For example, 
an objective at level 3, *Teople Inybived," could 
be to involve a certain, nuiriber of <:liehtele having 
prescribed xharacteristics/:^laceinent-9f **P'' on ♦ 
^ the sloping line at the left of each level, is shown' 

by a dot. The height oLthe dot'c^posite "P'^ 
. indicates the magnitude of . the objective. That is, a ' 
: dot. representing an objective to reach 200 clientele 
. would , be placed higher than if the objectiye were 
^ to reach y)0 clientele. The staircase ot objectives 
reaches toward solviflg^t tlie seventh level) 
some overall problem"7of clientele or the larger * 
society. Placement, ot flot^in figur^^^ 
^;of iUusfration. H«^^ levels 1 through 6, a / 
" pasis foiAsetting ;dl:>^^ is* their sufficiency to 
' move to th^ fleit higher lev;elts) and, finallyrto Ae 

- desired, end-results. . . * / ' 

Kgureig abbreviates, two dimensions or broad 
criteriayt each level; speofic samples of these and 
' other .criteria are provided' befow: ; ' 

1* At the level; criteria are within^^^ 
^(objectives) to allocate certain lands and amounte 
of resources to a program, su'ch as: 

: . • Time of paid sta& and/volunteers (e.g., "five 
V full-time equivalents per year wiU be expended 

- on-a consumer education prbgraci") . ' 

^ Staff qualific.ation---^paid and volunteer , (e.g.,. 
^ *^all program assistants to be recruited must be ' 
Obpinion leaders' "i. - ^; v ^ - ' 



2. At the activities le\e\, criteria are, within plans 
to perform, thrpugji, the above inputs, a certain 
number, of sjpecified activities in ordar to induce 
education, spch as f , * 



' •^^GoUecting- affd preparing' educatignar inat^rials 
^ *^assist volunteers ;in planting 20 plots to 

demonstrate researdi. findings"). . 

• Publicizing programs te.g., "publish five.n^ws- . 
> pappr notices of .'environmental activities" ).„ 
. V • ..Tfansmit^ing subjecft matter cojiteht thrpu^ 
. masg^,tnedia> Meetings, and other ^evehts <e.g:, / • > 
Y "^schedule fiy6 shpwings of a video-tapVon<how to 
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Figure 2. >• < 
ICEFORPROC 
fans Compared With Achievements 



HiERARCHY OF EVIDENCE FOR PROGRAM EVALUATION 
Plan 
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7. END RESULTS 



6. PRACtiCE. CHANGE 



5. ICASA CHANGE 



4. REACTIONS 



2.;AeTIVITIES 



1. INPUTS 




* / ULTIMATE OBJECTIVES; 
"Side" Effects 



Individuatlnnovation; 



Slructurail Change 



irection and Extent; 
Duration of Change 



Interest in Activities; 
Acceptance of Leadership 



PEOPLE INVOLVEMENT^ / Number and Characteristics; 
PEOPLE INVOLVEMENT / . continuity and imensity 



EdUcditional Methodology; 
Subject Matter Conveyed 



Time Expended^ 
. Staff Qualification? 



KEY ^ 

P = Planned (Objective) 
A = Achievement 
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3. At the people involvement leve}^ criteria are 
; within plans that certain types and numbers of ' 
persons, groups, cu: qommmiities wM^ be involved in 
the activities, suclr as: 

• Number of participants in>^nts, tours, meet- 
ings, or clubs (e.g., "2,^9e-6i^ore:^ 
members will be enrolled in Jivestock projects!*). 

.- • Psychological and socioeconomic characteristics 
of participants (e.g., "at least 90 percent of program 
clientele should be low-incomel')- ^ 

• 'Continuity, frequency, and inteilsity of face-to- 
^ace or mediated interaction between clientele and ' 
Extension , (e.g., "SO percent-of new pflScers of 
community development councils should attend 
leadership training ^meetings"). 



/4. At the reacr/ortj level, criteria are within plans ' 
to obtain certain reactions to involvement^in activi- 
• ties, in terms of: * V- ^ , 

■■-1 • . ^■. 

. \ • Interest id educati<)nal events (e.g., "there 
should be a minimum of ,75 percent p(^^ve reactions 
to topics chosen for discussion at child develop- 
ment meeting^"). 

•*^Acceptance of persons leading activities (e.g., 
'leader of soybean marketing meetings should be 
rated, as 'highly competent' by two-thirds of those 
in attendance'*). . . 

5*, At the KASA change level, criteria are within 
plans that certain knowledge, attitudes, ^skills, and 
•aspirations (KASA) will 'ensue from participants' 
engagement in program activitieis,^ including: 

• Direction (content) and extent of KA? A 
change (e.g., skyis— "80 percent of homemakers, 
rather thin the present 10 percent, to be able to 
.suitably^ arrange furniture in their respective 
homes"). . 



; ■'At all levelSi but especially at level* 5, *6, and 7, the 
question of whose objectives— Extension's or clientele's—- 
^ are involved may become an issue. The degree of consensus 

on objectives at these levels will depend on the adequacy 
''^^f Extension program planning: See Stalte, Robert E..;The 
* ^ Countenance Educational Evaluation/'' TeOqliers College 
Record 68 (April 1967), pp. 523-540. \ 



• Durability of any KASA change (e.g., knowl- • 
•edge— -'*'95 percent of farmers to recall sources of 
safety rules for handling pesticides one year after 
learning* about them"). . / . ^ v /V 

• Intensity, of attitudes to be accepted (e.g^, "all . 
youth in the Qitizenship seminar should 'come to 
^condemn very strongly the neglect to vote**)/ 

'f • Height of aspiration (e.g., "each couple repre- ^ 
sented in the family resource management workshop - 
should decide to prepare alegalVui within ! 
moqth after the close of the workshop"). V ' 

6, At l^e practice change level, criteria are within 
plans for certain changes in individual practices, 
technology, and/or social structures. These conse-. 
quences of KASA change are in terms of: 

Individual 'innovation and adoption (e.gV, "80 
percent . of farqiers to 'adopt new, superior variety 
, of wheat within 2 ^years of release''). 

Collective (structural) change (e.g*., "25 per-, 
cent of communities to estabjish land-u;se planning 
boards during each of 4 succ^sive years''):. 

"Individual innovation" is distinguished from / 
"structural change" in that the latter refers to / 
change in social relationships, laws', and institutions, ' 
including associated physical facilities. For example, 
if a solid \yaste disposal- systeiri is creat.ed in a 
• county, a Nvithin thalt county is "changed. 

7. At^the e/7rf r^^i^/i^vel, criteria ar^ " ' 
that certain effects Will be achieved through prad^ 
.change. These plans are called w////?/^/^ objectives 

and emphasize the preventipn^ checking, reduction,^ , 
or solution of overall problems of:* ' 

•^dividuals (e.g., :"one.-thirdr of 'isolate* youth * 
^attendihg^camp to gain incregsed selt and peer-^' ' ' 
acceptance"). 

' • Groups (e.g., :^the community wiU increase to 
5 percent its annual rate of real economic growt^"). 

REACHING PROGRAM OBJECXIVES . 

Figure 2 shows that actual outcomes or achieve- 
ments, "A" as well as objectives, pertain to each * 
of the seven levels discussed above: The height of . 
the dot opposite each ''A" shows the magnitude of 
actual outcome. . . i 
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Figure 2 also shows a variety S^[fpossible«Telation- 
ships between "P'^ and^^'A';. If the "A" is above Uie 
*T," piore ha5 been attained than planned, as shown 
at level 1 (e.g., seven, rather than the intended 
five, full-time equivalents are expended on the 
.program).'lf "A" is below >*P," less has t>een ac- 
complished than^pianned, as at level 6 (e.g., over 
a 4-year period -only 40 percent, rather than jfco 
percent, of communities establishecJ land-use 
planning boards). v^Of course, if plans Have been 
exactly^ttained,' "P" and "A" are the same, as 
depicted at level three (new officers of county de- 
velopment councils reach the objective qf 8() per- 
cent average attendance at. leadership training 
meetings). . > ' 

There are many, factors \yhich enter into value . 
judgments of programs. However, in general, the 
more nearly the objectives of a program are 
reached, the mor& positive the judgment of the pno- 
gram, i.e., the higher the Value assigned tp the- 
program. In turn, the more a program is valued, 
the more likely it will be continued, intensified^ or " 
broadened (uifless need for the prdgra^^ 
' lessened due to the program's success or to other 
factors). . " ' , 

Before continuing, it should be acknowledged that 
comparing objectives and achievements is by no^ 
meafis tfie only approach to evaluating Extension 
programs. Evaluations of program, impact may . * 
be based oh the entire array of program effects, 
wheflier or not related to program objectives. 

. "Side" effects 'may occur' at any output level of- 
the hierarchy, but apply especially to level- 7. 
"Side" effects are unintentional and usually tin- 
expected 9nd may be beneficial or harmful. For ex- 
ample, new. industry obtained by a community 
through Extension's assistance may alter establisheC 
social relationships in unexpected vvays. Other ap- . , 
proaches to program evaluation' include, comparing 
program, objectives and accomplishments with 
the mission of Extension as ah agency.^ 



* Steele, Sara M., Six Dimensions of^ firogram Effective- 
, ness\ Madison, Wis., ^ Program and Staff Developmentf 
University of Wisconsin-Extension, 1972. Also see Stake," 
Robert E., op, cit. « 



SELECTION OF LEVEL OF EVIDENCE ^ 

As /previously ou dined. Extension programs 
usually have— explicitly oe* implicitly — objectives aitl . 
sperd or all levels of the hierarchy depicted in flgurev; 
1. At which of the seven l^els should' evidence / 
if program accomplishments be obtained in evalu'at- 
ing Extension's effectiveness? Guidelines A, B^C, \ 
ana^ are offered* to help answer this question. 
These'^mdeUnes, as well as others in the paper, aire . • 
offered qh the jba^ of experience and logical 
plausibility. AlthpugKfli^^ h^ve not been - 

tested Systematically, theyare^^^vided in order 
to organize thinking about formal evaluation and 
fo lead 'toward: cumulation of tested principles about, 
evaluation itself. ' * • ' 

Guide A: Evidence of program impaet he' 
comes stronger as the hierarchy is ascended, 
(Of cpursev such evrdence may, indicate attainment, 
or lack of attainment, pf objectives;) Guide A states^ 
in effect, that evidence at 'the two lowest levels 
provides little or no, measure of the extent to which 
clientele benefit from the ^program. 

Level 3 merely provides one way of me^^uring i ; 
possible .opportunity for education to- occur. 
Evidence at the "people involved" level may sug-» 
gest the extent to which some kinds of benefits are 
being jeceijjgd by participants.. However, Evidence 
at this level (e.g., participation rate) doei not 
necessarily indicate progress toward ultimate pro- 
gram objectives: high participation may occur for 
some reason um^^elated to the benefits intended to * * 
accrue from the program. 

Ascending to the fourth level,*'."reactions^'*can 
rovrde somewhat better confirmation of whether 
'given activities are helpful as intended. vBut such 
[evidence indicates less satisfactoftljyftah evidence of 
KASA changes the extent of progress toward tilti- 
■ . mate.vpro^am objectivcis: KnbwleHge, skills, etc:/ to > 
^be,acquir^d are frequently considered as merely 
^'stepping stones" to adoption of more desirable . ' 
patterns of behavior, although there\re differing 
philosophies on whether practice chiange is always ' 
necessary to successful Extension education. 
Practice. change assessErtent is desirable when pro- 
. gram objectives include patterns of: (a) utilization 
or Application of new knowledge and skills; (b) . 
"expression of changed attitudes; and (c) follow- .: : 

11 ■ ' r 



through oh aspirations,^^ . 
ments.; r , , ' - ' ; , ' \ .. ■ ': 

Finally, assessuig practice change is usually /V-. 
. quite^ apart from assessing accomplishnient* of ; 
-inatd program objectives, Extension is often held 
accountable for the extent to. which it is contributiiig * 

• to^ solution or checking of overall probfcns of • 

• clientele dr the society. Therefojp, ideal evaluation 
. of impact of most Extension prograips would. , 

pi-obably 'be in terms of whether desired end ; " 
results are achieved, plus assessment of^ any sighif- . 
icant side effects. " , ' ^ ^ 

However, a reasori for infrequent assessment, 
of iihpact at the top levels of ifae hierarchy is set i 
. forth in Guide B: The d^culty and cost of 1 
obtttining evidence on program accompltshn 
fnenis generally increiises as the, hierflrc^^ ' v 
is ascended. Evidence .within, lovj/er levels of t}ie 
' hierarchy provides Uttie indicati of impact bu^ is 
comparatively inexpensive and easily gathered. ' 
_ As the Ijjprarchy is climbed, difficulty and resources 
" ■ required to measure actual program outcomes • 
generally increase, due to: J>):increasingly greater 
difficulty in setting precise objectives as guides id 
obtaining accomplishirient data — exclusions of 
- ; alteqiate objectives withia a le veLare more dliffioult 
to justify as the hiyarchy is ascended; (b) increas- 
ingly scattered sources of evidence— Extensiori 
clientele often apply sepSirately what they learn • 
. through participation in vgroup Extension activities; 
(c) . increasingly greatej; tfane-lag following program ' 
« activities— practice changes and end'resuhs may 
occur months to years after Extension activities; 
and (d) increasing difficulty of separating Extension 
accomplishments from accomplishments by other 
\ sources of change-^i.e., the higher in the hierarchy, 
^ the toore chance, that some agency, or a cominunica^. ^ 
tion source other than Extension, had a role in 
bqnging about any. observed change. / 
.. Guides A and B both assun^e e vidence^oL com- 
parable quality from one level to another; These , 
Tmd^^( A) evidence of impact becontes stronger . 
in asc^ing the hierarchy, and (B) more resources 
are required to collect evidence of accomplishment 
yvitNn^higher levels— SLve advanced only soAong 
as the quality of evidence remains constant from , 
level to level The quality of the evidence is dis- ^ 
cussed later in this ^publication. . y ' 
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.^^ PYRAB^ID oi^ EVIDENCE FQR 
PROGRAM EVALUATIpNv i ^ 

Figure .3 *depicts4 pyraihid which-'guicfts toward 
r the advantages of assessing a program at^ieVerat/ ^ 
levels of the Hieu-arctiy, inplndmg th^^to^ 
^ Figure 3 cumulates previously ;di:^cussed' levels of 
evidence in proce^ing from' Ev^encejClusiers I . 
to VII. Cluster I, simply the "input^' leyel, consti- 
' tute? an underlying coniponent, of all the other v 

- ctusters Of evideneeV Clyster it adds a second level, : ; 
"activities." ^ese» two levels jthemselve^s constitute 
"buikling blocfe" fcfr Ouster HI, and so on. . . - 
^'^^Cj Evjjiuation^ are strengiheheSL < 
inmExtension programs at seiferal levels of 

'^hierarchy including tlw inputs leveL Tliis 
ie is advanced for three reasons. • * 

rst, dlqng with other agencies. Extension is 
\asked if^reasipily to . report ^degree of output 
(le\^els 2 through 7) in relation to inputs or costs 
{level 1 of the hierarchy) . This entails analysis 
of program de^very efficien6y and; of cost effective- \ 

- ness or 'cost beneftts.'^ Clusters with higher numbers' 
- ' ("highy clusters-) grd^idfe for anallysis of program . 

cost in relation to_jttKtiveness criteria closer to 
problem solution 7leyel T} . . ' . ' ' . . / 

^ The second T^ason far Guide € is: the greater ^ ' 
the numh^er of program objectives ^hown to be met,' 
including thgse at varying levels, :the b^^^ ' ' • 

evidence of effectiveness. For example^, evidence of * 
intended.knowledge change improves certainty that 
clientele adoptions of recommended practices 
were made for the correct reasons or because of 
wliat clientele learned through parMcipation in 
Extension. 

A third reason for obtaining evidence at two or * 
mare levels 0^t he hierarchy is to check. on how 
far the program' has proceeded toward reaching Us 
ultimate objeGtives. A .program may^fall short *of 
inducing practice changes, but effectively induce 
intended KASA changes i external constraints may- 
pre vent Extension clientele -from putting into 
pracUcle 'khowiedge, attitudes, skillsj and aspirations 
acquireB through participation in Extensidii pro- 
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^ ^^'podi, Tony, Phi|lip'F6J)in, and \rwiri Epstein,' 5ocw/- 
Prosrani Evaluation, Itasca, fil., F. E. Peacock, 1971; 
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, Figure 3. 

A PYRAMID OF EVIDENUk . 
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Objectives achievements ar^ assumed, as represented in Figure 2. 

■ ' . *'13 ■ 



grams. Simijaily, objectives Jor practice change 
may have been achieved,, vi^ithout sufficient time 
haying elapsed for clientele to realize the envisioned - 
.benefits from the practice. , 

High clusters of evaluative evidence should be 
""selected to the degree that ^sources for formal 
evaluation are available, as higher numbered clusters 
provide stronger evidence for program evaluation. 
In the higher clusters, one or. more of the levels 
may be omitted, in line v/ith the purposes or con- 
straint's of compiling evidence for evaluation. 

The paper to this point may be partly summarized 
and also related explicitly to the chief purpose of 
prbgram evaluation, by stating Guvde^D: The higher 
the cluster of evidence for program evaluation^ 
the more useful the evidence for making 
decisions on present and future programming. 

EVALUATION CRITERIA AND 
QUALITY OF EVIDENCE 

Assessing program effectiveness generally requires 
specific criteria which can provide a^basis for 
. measuring the extent to which program objectives, 
have been attained.. 

Criteria withih program objectives are gener- 
ally definitions or subdivisions of objectives at 
each level of the hierarchy. Criteria are a primary 
basis for selection' of eviderifce as to the extent of 
accomplishment of objectives. For example, if the 
ultimate aim oi a program (level 7 of the hierarchy) 
is to achieve "desirable land-use," how would 
"desirable land-use" be defined? Would it be 'defined ^ 
in terms ofj^'rade-offs among preferred (a) "living 
space," (b/ "population growth," (c) "economic 
growth," i^id (d) "environmental status"? If so, 
how would (a), (b), (t), and (d) be defined? 
Repeated subdivfsion of the Components of (a),f(b), 
(c), and (d) would continue until criteria are 
sufficiently specific and clear to guide the selection 
of adequate evidence on the extent to which ultimate 
aims of the program have been achieved. 

The process of defining specific criteria for * . ^ 



Criteria to evaluating program, impiact may be unre- 
lated to program objectives. For example, criteria may be 
I'^sed on philosopfiical, ethical, or personal considerations. 



ev^uation is essentially one of moving from broad 
to specific objectives at eath level of the hierarchy. 
Therefore, planning for.obtaining evaluative ^i- 
•dence can and should occur simultaneously with the 
process of preparing mnltiyear gmgrams, annual*" 
plans of work; and leuiiiiug irfl 
• Guide E: Evidudtion is Hits ( igthened to the 
extent tfye specific criteria ^ valuation are 
defined^ prior to conduct of the Extension 
program. Specific criteria aie needed in order to 
obtain quality evidence on degree of attainment . 
of jprogram objective's: (a) prior to program activi- 
ties ("benchmark" evidence), and, (b) following 
stich activities. Early timing in planning for evalua- 
tion can clarify program objectives and, thus, 
also strengthen the planning and conduct pt Exten- 
sion programs. Timing in obtaining evidence will 
be discussed in some detail later. 

Evidence on the extent of accomplishment of 
objectives may vary in quality. Variation in quality 
of evidence is often referred to as "hard" versus 
"soft" data. Data; (i.e., observations) are "hard" to 
the extent that they are vaHd, representative, and 
quantifidd.^^ Figure 3 indicates that "soft" or- . 
"hard".dfita (or both) may be collected at , each 
levels of the hierarchy^ Jt should be empliasized that 
_ **hard" and "soft" data constitute a continuuiii;. 
a dichotomy^ is depicted simply for the sake of . 
convenience. ' . 

The degree of "hardness" of data actually selected 
d'cpejids upon trade-ofTs between ideal data for 
the evaluative purpose at hand and the resources 
available. Hard data are usually ideal; however^ 
'*hard" data are also more expensive and difficult to 
obtain and' should be collectccl only when the 
benefits to decisionmakings anticipated from superior 
evidence clearty outweigh the costs of obtaining 
'such evideii^, ^ 

There are many situations where '^'soft" data on 
degree of accomplishment of objectives are all that 
can be obtained; tot example, program participants, 
and especially nonparticipants, are often unwilling 



" For an Vntroduction to validity, quantificatioji, and 
representativTbness,- see Selltiz, Claire, Marie Jahoda. Morton 
Deutsch. and Stuart W. Cook. ResearcAUetho djiJiLSnclnl^ 
Relations, New York, Holt, Rinehart and Winston^ 1961. 



or unable to be observed or to respond to instru- 
nrlehts which require detailed answers and extensive 
time for completion. v 

Figure 4 illustrates ftie three principal dimensions 
of "hard'* versus "soft" data. Observations are 
valid to the extent that they truly reflect the charac- . 
teristics of individuals, groups, or situations undet 
. study. For example, regarding the measurement 
of knowledge (level 5), validity of responses by 
workshop participants to the followiiig question^ 
riaire item would be rather uncertain: "Please 
indicate whether you can recognize;* potassium 
defici^ency in wheat plants: (1) *yery confi- 
dent I can,' (2) 'fairly confident I can,' 

(3) , — 'not sure I can'." (A participant's selec- 
tion of one of the three responses could be observed 
and, therefore, be considered as data). A wholly 
valid measure of participants' actual knowledge 
would entail direct observation of the degree to 
which they can, in fact, accurately identify potassium 
deficiency under given conditions, such as develop- 
mental stage of plants presepted, etc. Precise 
definitions would be needed^ j6 specify observable 
actions indicating correct recbgnitioa af potassium 
deficiency ' ^ ■ 

Nonvalidity of data may arise from several 
sources. For example, awareness of program evalua- 
tion by participants may causp them to speak or 
act as they think they are :expected tp for the sake 
of the evaluation, rather than in ji6cordance with 
their own inclinations. Lack of validity ihay also 
arise from faulty instruments'^of observation, from ^ 
' observing too small a range of tictions by Exten- 
sion participants, and from perceiving participants' 
actions inaccurately due to personal bias. 

Validity of observtaions is demonstrated by the 
extent to which they are consistent with Other ' 
releyailt evidence concerning characteristics of 
individuals, groups, or situations under study. 
C^ide F: Evaluations are strengthened to the , 
extent that validity df observations has been 
demonstrated. 

Although true differences in characteristics of 



units may h& observed as differences (validity), the 
question as to amount or degree of difference remains. 
This poses the dimepion of quantification: The 
degree of difference^ay be shown by the assign- 
ment of numerals to represent quantities. Thus, 
quantitative data indicate How much difference 
thc|||k in individuals or structures which are ob- 

Sii^^ : uidi participants of a tour of feed lots 
r^^ n^M^- o rate various displ^^ed practices to 
miiVii environmental poflutibn as "acceptable" 
or "unacceptable" for their own pse (a measure of 
thejr attitude toward each practice). Such a rating 
^ does not permit measurement of whether one 
practice acdeptable to the individual exceeds his 
acceptance of another. The participant's responses 
could^be quantified by asking him to rate each 
pollution control practice on a scale of "zero through 
IQ". "Zero" could represent "totally unacceptable," 
and "10," "totally acceptable," with varying degrees 
of acceptability represented by nuAbers 1 through 9. 

The third aspect of hard data is that of represent- 
ativeness. Representativeness is the extent to 
which pbservations concerning individuals, groups, . 
or situations under study apply tp some total poi)u- 
lation of individuals, groups,-or situations. Repre- 
sentativeness may be dbtained by conducting a 
census or selecting a representative sample." In pro- 
gram evaluation, a census obtains information 
froni (or on) all the actual or potential program 
participants. A representative sample may be chosen, 
so that the information t)btained corresponds 
closely enough, for the purposes at hand, to com- 
parable census findings. Every tenth recipient of a 
consumer economics newsletter might be a suf- 
ficiently representative sample for the purpose of 
evaluating the newsletter. " 

The weight given to an evaluation in making a 
^ program decision should depend upon hardness^ of 
the e^dence. Guide G: The harder the evidence 
for evaluation^ the more an ^valuation may 
be relied upon in program decisionmaking. 
Table 1 shows examples of "hard"^ and "soft" data 
at each level of the hierarchy. 
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Table IV 

Examples of ^Hard" and ^^Soft" Qata in a Hierarchy 
' of Evidence for Program Evaluation 



vamplesV 



"Hard" data 



*'Soft" data 



7. End results 
6. Practice change 
5. KASA change 
4, Reactions ./ 
3, People invoiver 

2, Activities' 



1. Inputs 



Trends in profit-loss statements, life 
expectancies, and pollution indexes > 

Direct observation o' of om- 
mended farm pr : - -.^ o /c' eries 
of years 



Changes in score, 
ures oiTknowledg 
and aspirations , 



v::l :: ed meas- 
iit:i' skills, 



Extent to which randor :.i:.iirvle of 
viewers can be rL-rr3C- rem 
watching a dem^ni:> \iz.on 



Use of social p 
based on recorc 
attendance, hoi., 
positions, etc. 

Pre-structured o 
and social prbce- 
pant observation 
audio tapes, etc. 

Special observat 
'Expenditures, as 

study 

V. 



icipidcsv : .-ales 
'(7ser. ns of 
. of le:^ ship 



Casual 'perceptions of changes, in 
quality of health, economy, and 
environment 

Retrospective reports by farmers of 
r use of recommended farm - 

t^ractices ^ - . ^ 

Opinions on extent of change in 
participants' l^owledge, attitudies 
skills, and aspirations 

Recording th6 views oT only those 
who volunteer to express feelings 
about demonstration 

Casual observation of attendance and 
leadership by participants 

1 ■ li^i^^' ... ■ 



.)f activities" Staff recall of }iow - r rJes were 
pairtici- conducted and the ca. to which 
0 and they were completed 



time Staff's subjective repo: regarding 

...d motiorf' time allocation . 



IpROXY MEASURES 

Extension frequently lacks' sufficient resources 
tq\ obtain quality evidence of its exteiff of . 
eflfectiveness,. especially at higher levels of the 
hierarchy. In such calses, mferences of the degree 
tpvwhich objectives are attained can be made if 
• proxy or substitute measures have been established." 
Proxy measures are based on research-tested 
relationships between the achievement of objectives 
ati lower and higher^evels of the hier^chy, e.g., 
between KASA ch^ge and desired practice change 
in >^ youth community developmen^programs : 
On 'the basis of such previous research, reaching 
a lower level objective in a program permits 
inferring or predicting attainment of a higher level 
objective. Of course, caution must be exercised 
as to\how far previous research can be generalized as 
a basis for assessing program effectiveness. 

Witii their more confined scope arid variation, 
demonstration and pilot projects permit coljection 
of "high" evidence clusters with resources 
comparable to those necessary for collection of ^ 
"low" evidence clusters on full-scale programs. 
An efficient strategy for an agenda of formal ' ^ 
evaluation is this: collect high clusters on pUot 
projects and, in so doing, identify within 
lower level^ of the hieriarchy proxy measures of 
impact. These proxy measures can provide a basis 
for interpreUtion of subsequent low evidence clusters 
collected on\any ensuing full-scale program. 
Similarly, if Extension can evaluate full-scale 
programs periodically through high cluster 
evaluations, then, between such evaluations, low 
clusters can be used to make inferences about 
achievement of objectives at higher levels of 
the hierarchy. ■ ' 

Through application of the above strategies, a 
schedule of evaluations can be designed to provide 
systematically over a cycle of years for efficient 
formal evaluation of Extension's programs or pro 
gram components. Guide H: The efficiency of 
program evaluation can be increased through 
studies which identify proxy measures. 



designs/ FOR IDENTIFYING 
SOURCE/ OF IMPACT 

Study, (|esigns suggest schemes for collecting 
evidence of Extension's impact. Designs vary in 
strength of scientific eviden^regarding the ^ 
. extent to/ which KASA change, practiiie change, ^ 
or end results Nvere brought about through 
Extension rather than through other sources of - 
change./ Of course. Extension often works aldhg 
with other agencie/and institutions in addressing 
problems. 

Thete is scientific evidence of :^xtension's impact, 
to the/ degree that evidence can exclude or • 
take ihto account other possible causes of 
achievement of program objectives (e.g., other 
programs, chance events, maturation of participants, 
eflfec^s of being observed or tested before the 
program, special motivation of clientele involved 
in J^Xtensiqn ctc.),^* 

i^uide I: A sjtudy^S' usefulness for program 
decisionmaking is mhahced to the extent that 
it tan identify Extension's degree of contribu- 
tion to achievement of program ob jectives. 

jlhe following are only a fewof the possible ' ' 
study designs. First presented the field experiment, ' 
which provides strongest scientific evidence 
of the degree to which observed change is 
produced through Extension. Other designs are 
{^resented fe^rder of their capability of identifying 
t^e degree which Extension contributes to 
observed attainment of program objectives. The 
iesigns are not necessarily limited ^^e vy^y i^ i 



vhich they are described below: e^^^ay^^)e 
/more or less complex in being adapted\tb 
/varying conditions. The designs are defined and 
f illii^tfated below to show a range of possibilities 
'for identifying Extension's contribution to ' 
change. Finally, the designs are described in 
relatipn to program olypctives, in order to" 
show their relevance to program evaluation as • 
defined in this publication. 



'^Wholey, Joseph S., John W. Sctinlon, Hugh G. Duffy, 
James S. Fukumoto, and Leona M. Voght, Federal Evalua- 

* 4ion 'P oHc y > An^Ov ^ — ^^^^^ — . ' ' '" "" 

Institute, 1970. 



" Campbell! Donald T., and Jiilian C. Stanley, Experi- 
mental and nf)nr:i.p:Yni*rit>^^iit^i n^J^^..^ » ji 



Chicago, Rand McN^lIy, 19^3. Also Stouffer, Samuel A., 
"Some Observations on Study Design," The American 
Journal of Sociology 55 (January 1950), pp. 356-359. 
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16 N THE FIELD EXPERIMENT . 

Two instances of field experimentaLeyaluation 
•studies 'in Extension are: (a) the nation^ ^pact 
evaluation of Mulligan Stew, a televised nutrition 
program for youth/^ and (b) a study of Extension 
training impact on managers of Iowa retail . V 
farm supply firms.^^ " 

The field experiment requires piaking the program 
available .to clientele selected randomly (through 
chance alone) from some audience. The' part of 
, the audience selected for no exposure to the 
program is the^ "control group." For example, 
farm and family Extension- aides could be assigned 
to disadvantaged rural residents in half of the 
^ counties of a State. These counties, selected 

at random,* would contain the program- group of 
-xiisadvantaged rural residents. The. other counties 
would contain the control group. Observations 
before and 0€t the program, activities within , 
both the *pro^am and control groups are usually 
desirable in field experiments. However, 
observations only after the iactivities are permissible 
in ^ne conduct, of field experiments and may be 
preferred under some circumstances. ; 

Figure 5 depicts possible observations in a^fieldy 
experiment.^^In figure 5, levels of the hierarchy in 
which no observiations are made are represented by 
^roken lines^f Obseirvations prior to program 
activities ("before" observations) are made 
simultaneously at levels 5, 6, and 7 in both 
the program and control groups. Th6 "situation 
or benchmark" in each group is the same, V 
V as shown by the identical location of "Ab" relative to 
, the sloping lines of levels 5, 6, apd 7. Turning 
now to "during observations," the action strategy 
/ and reception actually occurred as planned, 
as shown by the coincidence of "P" and "A" at 
levels 1, 2j 3, and 4 in the program ^roup. 
"After" observations of KASA, practices, and end 




results are made as soon as it is reasoqrable to 
expect that the intended changes at these three 
levels have occxir red. ' c 

The interrelationships among objectives and 
observed' achievements ^h^!j[n in figure' 5 suggest an - 
effective Extension prpgram. First, "A" reaches "P" : . 
in the four top levels of. the /yro^riom group. 
Secondly, although each "A" in. the. control 
group is hi^^er than in the "before" situation, the rise 4 
is less than thf^rise of the corresponding program ' 
group "A." The contribution, of sources of 1 
change .other than Extension is shown by comparing 
the "before-aftep" observation within thejcontrol 
. group (Ab compared with A), A "significance" test 

can ^auge the odds that any greater increase in 
' program grpup achievement over that of <he 
* conrol group was brought about by the presence / 
of tne program rather than by uncontrolled 
•^factors or chance? ' ' 

The field experiment should he used when it is 
essen: al to have maximum certainty about the ^ . 
extent of Extension program impact In many , 
situations,' the field experiment is unattainable ■ 1 ) 
because of complexity or cost, or undesirable ^ . / 
because of ethical or political considerations. j| 
Una. such conditions, it is necessary tj settle '| 
for ciL'signs which provide evidence less ( 
conclusive of Extension's impacts . / 

■ ■ ■ - ' . ■ ■■ /.;■ 

^ MATCHED SET DESIGN / 

The comparison set design is similar to the 1 ; • 
field experiment except that program availability; 
^t6 a portion of the potential audience is on 
other than a random basis. Rather, a prograni group 
(set) arid a comparison set are usually selected ■ 
on the basis of Jheir similarity. For exaimple, 
(a) a study in New York State^^ compared 
progress of farmers in an Extension farm 
management program and progress of similar 
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nonprogram farmers; (b) a Maryland study^^ 
compared ■ progress of families in an intensive ' 
nutrition education program with , the progress 
of.families who were friends of program families, but 

•\yh<y V -not thcmsclvcs program participant^; 

till.vr \ > the iui. I oi a community improvement 
program in Kentucky was studied through 
matching program 'and nonprogram communities. . 

The basic limitaticn of the matched set 'design 
is that matching car be only partial, and not 
complet^.21 xo ^^^j^^ thW matching is , 

iifconiplete,ihe matched set te^^ 
identify accuratelv Bxtension'si:ontribution to 
change (as cqmpzrsd to other sources of change).. 
The matched set iiziisign does riot provide for > ' ' 
statistical .tests to -etermine the ^odds that 
extraneous factors are responsible for any greater . 
change in progran set "A" than in coihiparison ' 
set "A." Stati^te:^ lechniques sucfi as co-variance 
anialysis or multipe regression can correct" . . ; 
partially for such extraneous factors, but cannot 
substitute fully for the rancjopi assignment in ' ■ ' 
the field experimnit. The miatched set design' 
should seek to identify factors in addition to ^ 
B^ctension which may effect fcliange,' so that 
at least these factors may be accounted for ' 
statistically in assessing. Extension's degree of 
contribution to accomplishnient of program 
objectives. ; 

TIME-TREND STUDIES \ ( 

These studies follow, clientele's KASA change, 
pracuce change, olr problem solution-over an 1 

extended (e.g., multi-year^ period.>Therei are 
two major variations of t^lis method. [The first 
is time-trend projection of preprogram data vs. ' 
actual observations after program implementation. 
Program , impact is identified as the -difference r 

"between observed "after" program conditions : 



arid projected conditions based on rates of chanf 
from time periods prior to the program.^^ Of ^ 

-(bourse, to account for Uk jnount of changed * 
which 'has occurred, U. )S necessary to 'ook for V 
plausible* explanations other than the Extension 
program. This, design' is appropriate when there -is a 
trend that seems likely to have cqiitinuH if 
the program had not been introduced ..^.g., rate 
of increase in aver^p^|mber of pounc: of » / 
mUt^perxdw-peift^i^^ v ■ 

A second type of 't^pfie-trend' study is one'xyhich , 
obtains repeated measurement pf clienele. 

'Progress relative to program objectives.^ A prime 
example of this designys thp national ^valuation 
of the Expanded Food arid Nutrition Education, 
Program. In t^is program, clients kre enrolled 
as program participants, thus facilitating observation 
txf their KASA change, practice change, and / 
degree of problem solution over, the length of. 
.their participiation. Observation' of self-reported 
food consumption has been made beginning ^ 
with^ entry' of the client into the program and every 
6 months tHereafter.2^ . : 

"BEF^ORE.AFTER'^i§TUDY 

This^design requires observations both before^ 
and - after an Extension program, as could be . 
shown by the m-ogram grov.p portion of figure '5; no 
comparison set or "cQntro. group" is used. 
The before-after "design has b^en-used in riiany 
Extension studies and is v/ell exemplified by the 
evaluation of -a. Texas.- Extension pro-am' for 
low-income^ famjers.^J^ V ; - * 

The "before-after" design tests only partially ^- 
the extent to;which any changes at. hfgher levels in the- 

hierarchy: are produced by Extension inputs, >^ 
activitiesj-etc' But, it is plausible that Extension ' 
produced part of' any observed "impact, to the 



" Green, tawrence W., Virginia Li Wang, and Paul H. \. 
Ephross, "A Three-Year Longitudinal Study of the Iq^act ■ 
of Nutrition A^es on the.Kno\vIedge» Attitudes and Prac- 
• tices of Rural^cnr ^lomemakers." Paper present/^fl to 
American Public Health Association/ Atlantic Cir , Novem- 
ber 1972. 

^ Street , Pml, The Appalachian-JCommunity impact ^ 
Project: Comparison Change Among the Adults and 
Youth, Lexington. Ky,: Cooperative Enension Service, 1972. 

"Alpxander, Frank b., "A Critique of Evaluation" 
Journal of Extension : (W^inter 1965), pp. 2tf5-212. 
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Hatry;,Harry P., llichard E. Winnie., ahd Donald M: 
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tute, 1973. r . ' . 

Economic '^'Research Service, »r^v Expanded Food an 
Nutrii . Education Program 1969-1973» Wasbingtoh, t .J., 

^U.s! L. apartment of Agriculture, '1975. . . 
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. Ediication for Loy, -lncorrc Farmers, College Station, Tex.. 
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degree that other possible source of KASA- change 
practices change, etc., may be niled out logically: 

• However, simple comparison of ''before" andf 
"after" program data.may be misleading due to, 
unusual or normal fluctuations such ai seasbnal^ 

'variations; V! ' 

Although the designs described above-^( ^ ) field 

experiment, (b) matched set design, (c) time- . 

trend'^studies, and (d), before-after stiidy—ar^ ' 
/desirable, implementation of these designs ' • 

• can be xumbersomfe, expensive, and difficult to 
conn51ete soon enpiigh to. assist decisiongiaking on 
future programming. 1' . ^ . " 
^TKe genera'l use of d£i.ta "bn the hard side" 

in the foux designs, above" accounts for much ". '' ^ 
of their expense and. time consumption. MdreQver, 
. as the designs above select evidence; increasingly high 
in the hierarchy: (^) the longer it. is usually ; ] " 
necessary to wait *"till the data are in;'V (b) * 
the more things can complicjate the study, such as 
attrition of program participants; ^nd (c) the ;^ , . 
more expensive the study is fo complete. ^ 
A frequent'upshot is^use of the designs below which' 
are less capable of controlling for "rival 
dxplanatiOns" (i:e., attribiiting 'observed changes 
to sources other than Extension). . 



THEiiSURVEjY; 

■ ; In ^comparison with experimental, m^tjched-set, * 
tinie-trend, and:;"b.efore--after" designs, Ihe 

. survey desi'gn requires fe\yer resources per ^ 
prograrh 'participant observed. No -"before" 
observations are made in the survey, which may -.^ 

. be depicted by the ;*aftprT; (and also perhaps by the 
"during") observations shovv^n in figure 5. * 
Surveys in program evaluation may compare ^ 

: Extension clientele and non clientele within 

" higher levels of the hierarchy. 

Or, the survey may compare at one poiiTt jn tim- 
achievement of program objecti^s 6y Extension 
clientele with different characterstics, including 



: ' ^ ■■■■ . ■ .. ■ -c..-' 

Hays, Samuels P,„ Jr., Evaluating ^development Projects,^ 
Paris, 'Imprimerie Bouciin. United Nations "Educatignal, 
Scientific and Cultural Organization, 1955. 

Rose, Donald W., A Comjmrative Study of Two Pat- 
terns of Cooperative Extension Or^^anizcition.in Colorado ^ 
ami Their Axsociati'Sfj wiiji Goal Arhievcmcjit, Jobh^atis- 
faction and Ciient^'le Satisfaction, ph. D. DissertatiS^i. 
.University of Utah, 1971. ' 4 - 
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"^igh versiisi. w degreel> of program involvement. - I/. 
JinJsuch a survey, participants i^ith a low degrf!^ of 
' involvement constitute a partial substitute/for a • 

''comparison set/" "'^ 
; Primarily because of lack of situatioi;tai^data ' 
prior to an Extension program, the" survey : " ' 
generally provides rather weajc cpnclusrons about 
the extent to which Extension, rather thaii other 
forces, produce^ any^pbserved^differehces between . ' , 
Extension cliefttek and nonclienteie. limitation 
to "such inferences frffli surveys j^iclude sel^-selectipn . 
as a participant in Extension 4rid the effect 
on survey oliservations pf any "drop-oqts"' f roiir 
th^ 5xt^nsion program. Even with coniplex , * o^fM- 
Statisticar analysis^ the surv^y^ Visually provides .- '^■ f 
limited^capacity to account Tor the-degfee to' ■ '. 
which Ex;fjension fjroduces achievementjOf' " ; 
higher- level objectiv^. ./I ' _ . / 

An: important use of the survfey ij to collect.* • 
data on pei-ceptiorts or opinions abont ^e activities / 
arid outcomes of Extension programs^'^^ A random ; 
. sample of opmions as to effectiveness of Exte / ' 
"prpgrams may ]ie evidence sufficient to meet ,'1 
•-:evaluative needs. ' , ' /, I 

Opinions may be obtained regarding a wide j 
variety of areas, such' as : ( a ), the extent to 
which Extension prograriToBJectives have been- . 
achieved; 'tb), the extent to which Extension : h 
and other actors, agencies, etc., have; produced ; ^ 
. given^utcomes; and (c). the degree to which ' 
Extension clientele are satisfied with Extension's-') f0 
^programs. NumeVous studies using this methodojiyj^v' ' 
have been conducted.^^ A ^modified form of . / 
survey is elicitation ofTeTfbspectijie reports on ; . -r 
participants' status prior to their program ^ " 
participation. These reports 'provijde a partial . j j 
substitute for ''before" measurement's/'' Retrospective ' 
reports :irc 2' -rally iess reliable, tn::a re^^onses 
reflecting present, except V^ere substanriati'ng 
records ^are available. 

Despit^^ its many limitations, the survey 'design ^ 

Dayie» Lynn, Terry' Patterson, Dorothy MacKerachey, 
and Richard Cawiefy. SHAPES: )^lutrnri J'yn^^^,p^frt17iaiicn - 
System, Torontp/Can., Onlario Institute •:;>r'Studi^s in 
Education, 1975. . ^ 

Oldham,' Nfafvin D., and Claiide^F. Bennett, A Con-' 
certcd Effort iklHural Developmei: : An- vsis and Evalua- 
(ion, Stillwater, Coopera^i^Extens: :n Se: ice, Oklahoma 
State University, 1975 " 
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lends itself to many evaluation situations. It 
is condparatively simple an(} flexible. The survey 
may be employed after a program is implemented, 
without die prior evaluative planning required 
by some other desigtis. 

Finally, compared in terms of th^ number of per- 
sons or participants included in study, opinion 
surveys are usually less expensive than are the : 
study designs discussed previously. ^ 

THE CASE STUDY . / 

. Case studies observe inierisiveiy one or only a 
few selected individuals, groups; or communities. , 
Observation may .involve examination of existing 
records, interviewing, "or participant observation, . 

Case studies seldom carry- the rigor or formality 
of \he preceding designs. They often use soft data 
(especially In ^e;rms of questionable representative^ 
ness) and seldom. employ statisticd analysis. In 
contrast to the desigi^^cussed above, few, if any, 
expfitit comparisons ^^^ade: ths case selected 
for study is cfompared only implicitly with other cases 
casually observed or remembered. 

The weakest form of the case study, as used in 
Extension evaluation, is the isolated "success stofy," 
which documents the progresfe of only one or 
several cli^tele. Such case studies provide weak 
scientific evidence of Extension's iiiipact in a ^ 
community, state, or nationy because: (a) even 
if data on each case' is valid, the cases may irt>t be 
representative of Exten^iojn clieii:ele, and (b) the 
question of how much progress-^ientele and poten- 
tiial clientele would probably have made without 
Extension'5 aid is usually not ans^^ *ered satisfactorily. 

Stronger case studies are those conducted by 



outside observers using their own perceptions of 
.program process and impact and drawing on the 
observations^of key observers.^^ . * ' 

' The case study can draw together many diversp 
pieces of information into a unified interpretation , 
and may provide important evaluative insights. 
Thus, the case study can provide leads regarding the 
conduct and interpretdtidn of studies which use 
more definitive desigtis. ' 

Table 2 summarizes major characteristics of the ^ 
six designs above. , . ^ 

USING AND APRRAISiNG ^ 
EVALUATION STUDIES 

Evaluations of ^ program effectiveness are utilized 
most fully if their implications for decisionmaking 
are noted explicitly. Guide J; VsefulrHBssj of 
evaluation reports is maximiss&d when 
include alternatives and recommendattqns for 
future program development Interpretation of 
evaluationfindings for decisionmaking should 
include ^ appraisals of the quality and completeness 
of the evaluation study. . ' - 

The collection, analysis, and use of evidence in 
judging degree of program effectiveness should 
itself be assessed for effectiveness. If acquisition 
add use of evidence on program impact is viewed 
as an "activity*' thrdugh "inputs," then a number 
of questions follow, based on the iiierarghy for 
evaluation presented in this paper. Examples of ' 
these questions are: "What has been learned by ' 

^ Niederfrank, E. L, Francis S. Mansue,^nd Chester R. 
'Smith, Helping New J erje^ turban Youth Help Themselves, 
New Brunswick^T^J., Cbo^i^e^fijtive Extension Service, 
Rutgers University. : • . . 



Table 2: 

Characteristics of Designs iqr Analyzing 
Impacts of Extension Pril 



Evaluation Observations \ comparison set Evidence can 

/ . design "Before" ' "During" "After" 4^ed\ Randomly assigned apply, broadly 

Field Experiment Maybe Yes 

Time 'trend /Yti: Yes 

"Before-After" , ......... Yer Yes 

Survey Nc Mayoe 

Case"^ study MajTJe Maybe 




Comparison set 



Yes 



Yes 

-Yes- 
Yes 
Yes 
Yes 

Maybe 
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the collection andanaly^s of fflata, in relation to 
degree of expected improven/ent in knowledge about 
program effectiveness?*' "Have program decisions 
been influenced by knowledge of program effe'ctive- 
. dess acquired through evaluation studies?*' 

Appraisals of evaluation studies can suggest needs 
for further program evaluation, or related analyses, 
to assist in specific decision issues. 

^ SUMMARY :A^|D CONCLUSIONS 

The major'^rpose of evaluations is*to assist in 
program decisions. Formal evaluations are worth ^ 
doing onlyjf they have a chance of., Meeting such 
decisions. ■ V 

This publication presents options and guidelines 
relative to: (1) selection of strength of evidence 
of Extension's impact and (2). resources required 
. for obtaining evidence. Selection of strength and 
expense of evidence on program effectiveness vairy 
with informational needs and resources of decision- 
makers. * . * , ^ " 

Selection of evidence for evaluation studies 
^ should be guided by the foUowing questions: 

1. Which /eve/^ of evidence for program evaluation 
are desired for decisionmakingtelative to program 
continuation, direction, size, methodology, audience, 
etc.? ' ^ ■■ . ' ' 

-2. How "/irar^/" should the evidence be, and what 
kind oi study design is needed to assist materially 
in decisionmaking? 

3. Are resources available to obtain desired level(s) 
arid hardness of data, and to implement the de- 
sired study design? 

4. If the answer to question 3 is "yes," then fine! 1 ' 
But if the answer is "no," then: \» 

a. Can additional resources be obtained ;to ac^r 
quire the needed evidence? If the answer is again 
"no," then: ^ 

Jpii-. Cp" f^ef"<g'^»^"nnirp..r^ cvidonco from g 
' lower level, softer evidence, or evidence from a 
weaker study design? . • > ' 



"^Bennett, Claude F., "How to Analyze Impacts of,. 
Extension Programs," Program (and Staff Development; 
Extension Service,' USDA. 191^ 



Adequate judgments of program value and ^ound < 
progBam planning decision^can be made only by. r 
comparing clear criteri^fetfe^fficient evidence 
regarding program accomplishments. } 
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